Search results

This chapter looks at the role qualitative evaluation can play in the external review of the Probation Service, the development of an evaluation framework for ongoing assessment and how it can be used to develop new elements of the service. How is this different from the use of existing data by the service?

This takes us to the kind of information that is used by the probation service, to make judgements about the effectiveness of its programmes and the impact on offenders. The main contention is that this is in the main quantitative data, and reports on levels of re-offending. The data is standardised so that it can be used to make comparisons between different types of sentence and criminal justice intervention.

Our contention is that this information is limited in what it says for two main reasons. Firstly, quantitative data tends to report on impact, that is whether an offender has committed another crime after engagement with the probation service, or whether there are patterns of behaviour between offences, individual circumstances and the likelihood or re-offending. In short, the data is a snapshot of whether an offender has changed his or her behaviour or not. However, this chapter will illustrate how quantitative data misses an understanding of how behaviour changes and why behaviour does not change. As a result, this leaves the service with a limited understanding of how it is working.

Secondly, the chapter will argue that the existing probation framework itself creates quite specific definitions of which data is relevant and which data is not relevant. We will give examples of narratives that offenders offer in group sessions that provide rich material about their lives, pressures and offence. But evidence suggests that this information is not used to inform programmes, re-assess and review an offender's own progression towards re-offending or a life without crime.

Questions regarding the effectiveness of criminal justice efforts to reduce crime have dominated social and political thinking in this area for more than a century (Bowen, 2011). During this time a number of philosophical shifts regarding the aims of correctional systems have occurred, fuelled typically by the prevailing political standpoint (McGuire, 2005). At the start of the twentieth century, policymakers in the United States and United Kingdom placed faith in the rehabilitative ideal and offender ‘treatment’-dominated corrections policies (Cullen & Gendreau, 2000). In this context ‘treatment’ refers to a range of interventions designed to alter the individual, contextual and social factors that sustain offending behaviour (Hollin, 1999). This remained the prevailing perspective for the subsequent seven decades until questions arose regarding the quality of ‘state run’ corrections facilities in the United States in the early 1970s. At this point, evaluation science was one of many influences on a change of policy (Hollin, 1999). Martinson's (1974) now infamous research synthesis, described by Glaes (1998, p. 713) as ‘a watershed moment’, provided politicians and policymakers with greater justification for changing the focus of corrections policy. Although arguably it was observers misreporting of Martinson's claims about the evidence which were most influential, rather than the review itself. According to Martinson (1974):It is just possible that some of our treatment programs are working to some extent, but that our research is so bad that it is incapable of telling. Having entered this very serious caveat, I am bound to say that these data … give us very little reason to hope that we have in fact found a sure way of reducing recidivism through rehabilitation. This is not to say that we found no instances of success or partial success; it is only to say that these instances have been isolated, producing no clear pattern to indicate the efficacy of any particular method of treatment. (p. 49)

What are the boundaries of a case study, and what should new evaluators do when these boundaries are breached? How does a new evaluator interpret the breakdown of communication, how do new evaluators protect themselves when the evaluation fails? This chapter discusses the journey of an evaluator new to the field of qualitative evaluative inquiry. Integrating the perspective of a senior evaluator, the authors reflect on three key experiences that informed the new evaluator. The authors hope to provide a rare insight into case study practice as emotional issues turn out to be just as complex as the methodology used.

In our public institutions it sometimes feels as though the tectonic plates of social and political contracts are shifting. Familiar coordinates are displaced or left stranded as we survey new territorial configurations and have to work out again how to find our way in public administration. The economic revolution embraced by neo-liberals and conservatives found its counterpart in a governance revolution in the very institutions that have always been designed to protect us from sharp historical and political lurches one way or another. Our public institutions – mostly coinciding with what we can call our professional institutions – which were once the filters of social and political change have become co-opted into social reform and are now all-too-frequently the conduits of that political change.

David Marquand (2004) opens his celebrated book on the ‘decline’ of public service by arguing (pace Tawney) that there is a fundamental tension between capitalism and democracy – the former dedicated to an essential economic inequality (competition); the latter dedicated to rights-based equity derived from civic ideals. His book highlights public sector institutions as the arena in which the inevitable tussle between the two is played out. The neo-liberal movement provides the ideological vehicle for reform of public institutions, designed to ‘root out the culture of service and citizenship’ (p. 2) – i.e., those civic ideals. At the heart of the struggle is professionalism, the combination of competence, judgement and principle, and neo-liberalism takes careful aim at it.

Much programme and policy evaluation yields to the pressure to report on the productivity of programmes and is perforce compliant with the conditions of contract. Too often the view of these evaluations is limited to a literal reading of the analytical challenge. If we are evaluating X we look critically at X1, X2 and X3. There might be cause for embracing adjoining data sources such as W1 and Y1. This ignores frequent realities that an evaluation specification is only an approximate starting point for an unpredictable journey into comprehensive understanding; that the specification represents only that which is wanted by the sponsor, and not all that may be needed; and that the contractual specification too often insists on privileging the questions and concerns of a few. Case study evaluation proves an alternative that allows for the less-than-literal in the form of analysis of contingencies – how people, phenomena and events may be related in dynamic ways, how context and action have only a blurred dividing line and how what defines the case as a case may only emerge late in the study.